science funding

=institutions =funding =research

 

 

Current science funding systems are widely considered unsatisfactory, but there's no consensus on what to do differently. Here are my thoughts on that topic.

 

 

some current issues

 

- Professors spend a lot of their time writing grants. In many cases, most of their time is taken up by grants and meetings.
- Grant distribution is slow.
- Grants are determined by average scores among a committee, so controversial proposals lose to more conventional ones.
- Grant reviewers want to fund the approach that seems most promising, and they don't know what other reviewers are approving, so redundant research on consensus approaches gets funded.
- In many departments in US universities, most graduate students are foreign nationals, and most of them leave after getting their degree.
- Journals charge high fees for access despite not paying reviewers, giving them enormous profits from the name and prestige of journals. Professors try to publish in those journals anyway, because promotions and grants are based partly on prestige of journals.
- Reviewers - being unpaid - often don't do a good job or take a long time.
- Publication is slow, partly because reviewers take a long time to respond.
- Faking research results is incentivized.
- Some reviewers vote against proposals then steal ideas from them.
- Some reviewers demand that their own papers be cited, because citations are another important metric.
- Some reviewers vote against research directions that could compete with their specialty.
- People trade citations for favors or other citations. This is now widespread.
- Because people who trade citations and otherwise exploit the system do better, they're selected for professorships.
- University labs tend to be little kingdoms that don't share resources or cooperate.

 

This is by no means meant to be a comprehensive list.

 

 

some existing proposals

 

Some papers have concluded that randomly distributing funding to proposals would work as well as the current system. However, if funding was distributed completely randomly, there would be more low-quality proposals. Most proposals have a minimum score threshold followed by a random lottery; that system has been used in practice, and works well enough, but this doesn't solve the above issues.

Some people are now proposing giving funds to professors who must redistribute those funds to other people's research but can do so freely. If this was used for most government research funding, it would result in large-scale corruption and patronage networks taking over research. Such networks take some time to develop, as citation rings did, but I expect any improvement would be temporary.

 

 

grant writing

 

Professors spend too much time writing grants. My proposal is:
All grant proposals for government money must go through a central system. Individuals are limited to 1 grant proposal per month. Professors can submit proposals for any students in their lab, but if they do so, funding is tied to that student, not the lab, and the student can go to another organization that has the necessary equipment and a history of acceptable results.

 

 

citations

 

I often look for information on a recent research topic, which leads to me often looking at minor scientific papers from the past 10 years. Sometimes I want to check for more-recent related results, so I'll search for papers citing what I read. It's common for the majority of those search results to be in the same field but not have any relation to the paper's topic. I suspect that most of those citations are traded or added as favors to eg reviewers or people in the same lab or university. If my perception is accurate, a large fraction of citations of papers in the past 10 years are not based on the paper's usefulness. Citation rings are a big deal, but I'm guessing that most questionable citations are traded less formally than that.

Many professors at US universities are already tenured based on traded citations, and those professors aren't likely to make the system work better. Many of the current stakeholders benefit from citation trading, so attempts at reform might only block competition to existing citation rings.

 

"Salami slicing" is another common practice. Instead of publishing one good paper, if you split it into minimal publishable units, you can get more publications and citations.

 

My proposal is:
Instead of using citation counts, make authors of papers list their top 3 citations, ranked. Governments would need to make authors do this for past papers, as well. Then, only count those 3 citations for evaluations.

 

That would:

- make weak and unrelated citations worthless
- make salami slicing counterproductive
- make it difficult for reviewers to demand useful self-citations

 

 

objective proxies

 

Students applying to universities generally take standardized multiple-choice tests, because scores on them are easy to measure objectively. If students trying to start an undergraduate or graduate program must take standardized tests, why don't professors have to as well?

One objection is that the standardized tests for students would be too easy. To get to a professorship, people generally did well on some standardized tests at some earlier stage. The resulting pool of graduate students is too large, especially if you include potential ones, but simply raising score thresholds isn't good, because the tests lose meaning at some point. In general, multiple-choice tests aren't very useful for distinguishing between test takers that are smarter than the test writers. For example, smart people taking IQ tests end up having to guess at the thought process of the test writers. So, good professor-level standardized tests aren't trivial.

A simple way to get more distinction between top scorers with a standardized test is to have more problems and make people solve them quickly. This is what IIT exams do. This selects for fast thinkers, and humans have a tradeoff between fast and deep thinking, so this selects against deep thinkers. (Someone probably wants to reply with a comment about John von Neumann, but fast mental math being a dumb person's idea of smart people is part of why von Neumann got credit for so many other people's ideas.)

Fortunately, there are already some more difficult standardized tests - for example, the International Math Olympiad is probably difficult enough to challenge many professors. However, most fields don't have something comparable, and math research is already one of the more meritocratic fields.

Another objection is that standardized tests aren't very similar to research or teaching, and more expensive evaluation than students get is worthwhile for professors. However, if the goal is to avoid quid pro quo outweighing merit, then some sort of system with automated and thus objective evaluation is probably necessary.

Harvard entrance exams used to test people on their knowledge of Latin and classical construction. Today, perhaps some video game would be suitable. But - what kind of video game?

Chess? No, chess is too *small*. Hiring researchers based on their skill at chess would be like hiring soccer players based on their rank in an arm wrestling tournament. It's better to recruit soccer players from sports with more well-rounded athletes, such as water polo or pentathlons.

On the other hand, when it comes to research, people need to specialize, so it's not good to rank people by their average score across many areas. It's probably better to let people choose a few things to be ranked on.

 

Some other things that aren't good for this:

- games largely about precise timing (eg Hades, Celeste)
- games largely about aiming at targets (eg FPS games)
- games that can largely be solved by memorization
- games that AI is better at than humans

 

The most suitable game types for this purpose seem to be:

- roguelikes
- builders with random challenges
- programming challenges

 

For example:

- leetcode
- DCSS
- Vault of the Void
- modded Kerbal Space Program, given a destination and time limit and set of usable parts, and minimizing initial cost
- Satisfactory, given a starting save and set of products to make and a time limit, and being scored on factory aesthetics by a panel of reviewers who look at save files
- Ludum Dare type competitions

 

One problem with both games and standardized tests as a metric for people is selection of and training for over-reliance on stated rules and systems working as intended. For example, in Kerbal Space Program, players often make complex staging arrangements that, if realistic system reliability was considered, would be bad. Ludum Dare as a challenge partly avoids this issue, especially if using Unity as a game engine.

 

 

Well, I'm talking a lot about video games and standardized tests here, but the point is just to have some sort of objective competition. Neural network research has been driven forwards largely by competitions such as ImageNet benchmarks. If something like computer chess competitions or protein folding or optimizing power inverters is relevant, then that's often better. The problem is, that's often not possible or very expensive. Computer game competitions have low capital costs and the bonus of being potentially entertaining to watch.